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Part  of  the  Dynamic  Systems  Program,  the  MSCE  Project  develops 
methods,  tools,  and  techniques  for 

•  Advancing  the  state-of-the-practice  for  risk  management 

•  Managing  assurance  in 

Multi-enterprise,  distributed  projects  and  processes 

Software-intensive  systems  and  systems  of  systems 

The  project  team  builds  on  more  than  15  years  of  SEI  research  and 
development  in  risk  management 

•  Continuous  Risk  Management  for  software-development  projects 

•  Operationally  Critical  Threat,  Asset,  and  Vulnerability  Evaluation 
(OCTAVE®)  for  organizational  security 
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Background 


Traditional  Approaches:  Projects  and  Support 
Functions 


Traditional  management  approaches 
focus  on  issues  directly  under  the 
control  of  projects  or  operational 
processes 

Various  functions  within  an  organization 
that  support  projects  and  processes  can 
increase  or  mitigate  risk 

•  Some  of  these  functions  are 
outsourced  to  third  parties 

•  Decision  making  is  usually  not  well 
coordinated 
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Distributed  Programs  and  Operational 
Processes 


Frequently  collaborative  ventures  with 
multiple  organizations 

Partner  actions  can  increase  or  mitigate  risk 

Distributed  programs  and  processes  are 
especially  vulnerable  to 

•  Conflicting  priorities 

•  Uneven  resource  allocation 

•  Complex  interrelationships 

•  Dynamic  conditions 

Typical  consequences  can  include  hidden  risks, 
locally  optimized  risk  mitigation 


Organization  A  Organization  B 


Organization  C  Organization  D 


unmitigated  risks,  and 
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Need  to  Establish  and  Sustain  Momentum 
Towards  Success 


□ 


□ 


LJ 


Momentum  Toward  Objectives 


Objectives 


Achieving  success  requires 

1 .  Establishing  sufficient  momentum  toward  objectives 

2.  Sustaining  momentum  when  stressed  by  events 

3.  Sustaining  momentum  when  circumstances  change 
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MOSAIC 


SEI  Mission-Oriented  Success  Analysis  and  Improvement  Criteria 
(MOSAIC) 

•  Is  a  management  approach  for  establishing  and  maintaining  a  reasonable 
degree  of  confidence  that  objectives  will  be  achieved 

•  Comprises  a  suite  of  assessment  and  management  methods 

•  Can  be  applied  across  the  life  cycle  and  supply  chain 


Strategy  Evaluation 


Design 

Planning 


Testing/ 

Integration 


Operations/ 

Maintenance 


Concept 

Exploration 


Requirements 

Analysis 


Development 

Activities 


Release/Production 


- V - 

Project  or  Program  Execution 


-A. 


Operational  or  Business 
Process  Execution 
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Focus  on  Outcomes 


Context 


t 


Success 


4 

5 


Failure 


Range  of  Potential  Outcomes 


t 


Traditional  Risk  Management 


MOSAIC 


Traditional  risk  management  is  focused  on  managing  potential  problems  or 
obstacles  that  can  lead  to  adverse  consequences 

MOSAIC  is  focused  on  managing  the  outcome,  or  result,  of  each  project  or 
business-process  objective 


Software  Engineering  Institute  CarnegieMellon 


Mission  Success  in  Complex  Environments 

©2008  Carnegie  Mellon  University 


Overview  of  Mission  Diagnostic 
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MOSAIC  Assessments 


Mission  Diagnostic  Protocol 
(MDP) 


Mission  Assurance  Analysis 
Protocol  (MAAP) 


i 


i 


Two  protocols  are  currently  defined: 

•  MDP  is  a  simple,  time-efficient  analysis  that  estimates  the  potential  for 
success  for  a  project  or  process  based  on  a  small  set  of  key  drivers 

•  MAAP  is  an  in-depth,  complex  analysis  that  determines  the  potential 
for  success  for  key  objectives  in  distributed  environments  based  on  both 
key  drivers  and  an  operational  model 
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Potential  for  Success 


The  likelihood  that  an  objective  will  be  achieved 


Excellent 


High 


Medium 


Low 

|  Minimal 


The  objective  will  almost  certainly  be  achieved. 

The  objective  will  most  likely  be  achieved. 

The  objective  is  just  as  likely  to  be  achieved  as  not. 
The  objective  will  most  likely  not  be  achieved. 

The  objective  will  almost  certainly  not  be  achieved. 
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Applying  MDP 


Positive  Conditions 
and  Potential  Events 


Negative  Conditions 
and  Potential  Events 


Driver  1  > 

Driver  2 


Driver  3 

> 


Driver  n  J 

\ 

Focus  of  MDP 


Excellent 


High 


Medium 


Low 

|  Minimal  | 

Potential  for 
Success 


The  potential  for  success  is  determined  by 

•  Evaluating  a  small  set  of  key  drivers  of  success  or  failure 

•  Applying  a  simple  algorithm  to  determine  the  potential  for  success 


=■  Software  Engineering  Institute  CarnegieMellon 


Mission  Success  in  Complex  Environments 
©2008  Carnegie  Mellon  University 


13 


What  Are  Drivers? 


A  driver  is  a  condition  or  circumstance  that  influences  the  outcome  of  a 
project  or  business  process 

•  A  success  driver  guides  a  project  or  business  process  toward  a 
successful  outcome 

•  A  failure  driver  guides  a  project  or  business  process  toward  an 
unsuccessful  outcome 

Each  project  or  process  has  a  mixture  of  success  and  failure  drivers 
influencing  the  eventual  outcome 

Drivers  are  used  to  estimate  the  degree  of  momentum  toward  project 
or  business-process  objectives 
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Consider  a  Wide  Range  of  Drivers 


Outcome 


/\ 


Design 


/ 

/ 

/ 

/ 
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Events 
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Execution 


/  m  w  i  jj  > 

\  iir  / 
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\/ 


Environment 


You  need  to  analyze  a  wide  range  of  success  and  failure  drivers 
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Generic  Set  of  Drivers 


1 .  Are  project  goals  realistic  and  well-articulated? 

2.  Are  communication  and  information  sharing  about  mission  activities 
effective? 

3.  Are  customer  requirements  and  needs  well  understood? 

4.  Are  organizational  and  political  conditions  facilitating  completion  of 
project  activities? 

5.  Is  the  project  plan  sufficient? 

6.  Does  project  management  facilitate  execution  of  tasks  and  activities? 

7.  Is  task  execution  efficient  and  effective? 

8.  Is  staffing  sufficient  to  execute  all  project  activities? 

9.  Are  the  technological  and  physical  infrastructures  adequate  to 
support  all  project  activities? 

1 0.  Are  changing  circumstances  &  unpredictable  events  effectively 
managed? 
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Evaluating  Drivers 


Question 

Answer 

No 

Likely 

no 

Equally 

likely 

Likely 

yes 

Yes 

1 .  Are  project  goals  realistic  and  □ 

well-articulated? 

□ 

□ 

□ 

Each  driver  is  evaluated  based  on  the  data  collected 
Probability  is  incorporated  into  the  range  of  answers  for  each  driver 
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Analyzing  Project  Drivers 


Yes  — 
Likely  yes  — 
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A  simple  analysis  provides  insight  into  the  potential  for  success 
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Managing  the  Potential  for  Success 


The  potential  for  success  is  the  likelihood  that  the  desired  outcome  will  occur 
The  goal  is  to  ensure  that  the  potential  for  success  is  within  tolerance 
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Applying  Mission  Diagnostic 


Applications  of  Mission  Diagnostic 


We  have  applied  Mission  Diagnostic  (MD)  in  the  following  domains: 


•  Cyber-security  incident  management 

•  Software  development  portfolio  management 

•  Software  development  and  deployment 

MD  proved  to  be  an  effective  in  all  cases 

For  each  domain,  we  tailored  the  MD  drivers 
and  some  of  our  techniques 
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Cyber-Security  Incident  Management 


We  used  MD  as  an  adjunct  to  a  detailed  functional  assessment  to 
provide  a  broad,  risk-based  view  of  the  response  team’s  potential  for 
successful  operations 

•  Identified  10  drivers 

•  Additional  5-10  minutes  per  interview  using  broad  questions 

Assessed  operational  processes  and  practices  used 

•  To  prevent,  detect,  and  respond  to  incidents 

•  For  various  types  of  events  and  incidents 

Method  was  transitioned  to  incident  response  team  experts  for  further 
use 
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Software  Development  Portfolio  Management 


Customer  wanted  a  quick,  risk-based  means  of  sorting  through  various 
software  development  projects  based  on  their  potential  return-on- 
investment  and  risk  at  different  points  in  their  life-cycles 

•  Identified  14  drivers  based  on  previous  successes  and  failures 

•  Conducted  face-to-face  interviews 
Transitioned  method  to  client  at  the  end  of  the  first  pilot 
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Software  Development  and  Deployment 


Used  MD  for  a  rapid,  high-level  assessment  of  the  potential  for  a 
successful  deployment  of  a  software-intensive  system 

•  Identified  18  drivers,  with  a  particular  focus  on  deployment  concerns 

•  Conducted  interviews  using  teleconferencing  to  keep  costs  down 
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Lessons  Learned 


Self-Application 


MD  assessments  can  be  self-applied 

•  Simple,  algorithmic  aspect 

•  Generic  set  of  10  drivers  is  useful  in  most  applications* 

•  You  do  not  have  to  be  an  expert  in  MD  to  get  actionable  results 


We  have  successfully  transitioned 
tailored  MD  assessments  to  customers 


*  Tailoring  drivers  does  require  some  expertise  and  experience 
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Number  of  Drivers 


Time-efficiency  is  a  key  aspect  of  a  MD  assessment;  keeping  the 
number  of  drivers  small  is  essential 

Between  10  and  15  drivers  will  generally  provide  good  results 
We  have  successfully  used  18  drivers 
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New  Sets  of  Drivers 


We  began  with  a  generic  set  of  10  drivers,  then... 

•  Tailored  the  generic  drivers  to  create  a  10  driver  set  for  cyber-security 
incident  management  projects 

•  Developed  a  new  set  of  14  drivers  with  a  focus  on  ROI  and  other  business 
concerns  for  software  development  portfolio  management 

•  Developed  a  new  set  of  18  drivers  focusing  on  technical  and 
programmatic  concerns  for  system  development  and  deployment  projects 
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On-Site  Interviews  and  Teleconferencing 


Usually  used  on-site  interviews,  requiring  from  10  to  45  minutes, 
depending  on  the  number  of  drivers 

•  On-site  interviews  can  be  more  effective,  but  are  harder  to  schedule 
and  can  require  additional  expenses 

•  Teleconferences  were  just  as  effective,  but  did  raise  issue  of  being 
unsure  who  was  really  on  the  other  end  of  the  phone 

•  In  cyber-security,  we  used  only  4-5  questions  to  collect  information  for 
the  10  drivers;  other  information  came  from  the  parallel,  in-depth 
assessment 

All  techniques  were  effective  at  raising  concerns,  strengths,  and  issues 
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Algorithmic  Analysis 


MD  assessments  use  simple  algorithms  to  calculate  the  potential  for 
success 

•  Does  not  require  extensive  risk  or  assessment  experience  to  use 

•  Basic  means  of  identifying  potential  for  success 

•  Results  are  sufficient  for  managers  to  determine  where  to  make 
improvements 

•  Provides  only  a  broad  view  of  the  potential  for  success 

More  complex/advanced  analyses  would  be  needed  to  provide  a  more 
refined  view  or  to  consider  alternative  outcomes 
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Outcome-Based  Scenario  Analysis 


For  software  development  and  deployment  projects,  we  borrowed 
outcome-based  scenario  analysis  from  the  more  complex  MAAP 
assessment 

•  Determined  minimal,  moderate,  and  good  pictures  of  success  and  the 
potential  for  each  to  occur 

•  Able  to  show  that  at  least  some  type  of  success  was  possible 

•  Requires  additional  expertise  to  identify  and  assess  alternative  scenarios 
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Useful  Complement  to  In-depth  Assessments 


When  used  with  the  in-depth  functional  assessment  for  cyber-security 
incident  management  teams,  MD  provided  a  useful,  alternative  view 
into  the  current  state  of  the  team  and  its  operational  processes 

•  Easier  to  understand  the  key  issues  and  risks  (1 0  or  less) 

•  Senior  management  quickly  understood  the  situation  and  what  was 
needed  for  improvements 

•  MD  results  were  used  by  senior  managers  to  deal  with  risks  that  were 
beyond  the  control  of  the  technical/project  leads 

•  Drivers  provided  a  more  effective  means  of  quickly  communicating  risk 
between  senior  managers  and  technical/project  leads 

In-depth  assessment  results  were  used  by  technical/project  leads  to 
conduct  localized  improvements 
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New  Areas  of  Research  and  Development 


From  the  software  development  and  deployment  project,  we  will  create  a 
new  assessment  protocol  that  blends  MD  and  MAAP 


New  Protocol 


Mission  Diagnostic  Protocol 
(MDP) 


Mission  Assurance  Analysis 
Protocol  (MAAP) 


i 


i. 


•  Working  with  different  layers  of  information,  responsibility,  communication,  and 
risk  mitigation  across  and  within  organizations  has  started  research  into  a  new 
taxonomy  for  success  management  based  on  conditions  and  events 

•  We  will  be  conducting  research  into  using  the  MD  as  a  basis  for  continuous 
management  of  project  and  process  risk. 
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You  Don’t  Need  Detailed  Assessments... 


...to  see  you  are  going  in  the  wrong  direction! 

A  quick,  efficient  assessment  like  the  MD  can  reveal  if  you  are 
generally  heading  for  success  or  failure 

•  Point  out  areas  that  need  to  be  improved 

•  Identify  general  areas  that  could  benefit  from  detailed  analyses  or 
assessments  (e.g.,  a  security  assessment) 


A  quick  assessment  of  your  current  state  can  make  you  stop  and 
think... and,  sometimes,  that’s  what  you  need  the  most 
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For  Additional  Information 


Christopher  Alberts 

cia@sei.cmu.edu 

412-268-3045  (Office) 
412-268-5758  (Fax) 


Audrey  Dorofee 

aid@sei.cmu.edu 

41 2-268-6396  (Office) 
412-268-5758  (Fax) 


For  updated  slides  or  more  information 

http://www.sei.cmu.edu/msce/ 

sei-mosaic@sei.cmu.edu 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 
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